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Why the Future Doesn't Need Us 


Why the future doesn’t need us. Our most powerful 21st-century 
technologies — robotics, genetic engineering, and nanotech — are 
threatening to make humans an endangered species. 
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FROM THE MOMENT I became involved in the creation of new technologies, their ethical 


dimensions have concerned me, but it was only in the autumn of 1998 that I became 
anxiously aware of how great are the dangers facing us in the 21st century. I can date the 
onset of my unease to the day I met Ray Kurzweil, the deservedly famous inventor of the first 


reading machine for the blind and many other amazing things. 


Ray and I were both speakers at George Gilder's Telecosm conference, and I encountered him 
by chance in the bar of the hotel after both our sessions were over. I was sitting with John 
Searle, a Berkeley philosopher who studies consciousness. While we were talking, Ray 


approached and a conversation began, the subject of which haunts me to this day. 
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I had missed Ray's talk and the subsequent panel that Ray and John had been on, and they 
now picked right up where they'd left off, with Ray saying that the rate of improvement of 
technology was going to accelerate and that we were going to become robots or fuse with 
robots or something like that, and John countering that this couldn't happen, because the 


robots couldn't be conscious. 


While I had heard such talk before, I had always felt sentient robots were in the realm of 
science fiction. But now, from someone I respected, I was hearing a strong argument that they 
were a near-term possibility. I was taken aback, especially given Ray's proven ability to 
imagine and create the future. I already knew that new technologies like genetic engineering 
and nanotechnology were giving us the power to remake the world, but a realistic and 


imminent scenario for intelligent robots surprised me. 


It's easy to get jaded about such breakthroughs. We hear in the news almost every day of 
some kind of technological or scientific advance. Yet this was no ordinary prediction. In the 
hotel bar, Ray gave me a partial preprint of his then-forthcoming book The Age of Spiritual 
Machines, which outlined a utopia he foresaw—one in which humans gained near 
immortality by becoming one with robotic technology. On reading it, my sense of unease only 
intensified; I felt sure he had to be understating the dangers, understating the probability of a 


bad outcome along this path. 


News of the future, now. News of the future, now. 
Get WIRED for as low as $5. 


Subscribe Now 


I found myself most troubled by a passage detailing a dystopian scenario: 
THE NEW LUDDITE CHALLENGE 


First let us postulate that the computer scientists succeed in developing intelligent machines 


that can do all things better than human beings can do them. In that case presumably all work 
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will be done by vast, highly organized systems of machines and no human effort will be 
necessary. Either of two cases might occur. The machines might be permitted to make all of 
their own decisions without human oversight, or else human control over the machines might 


be retained. 


If the machines are permitted to make all their own decisions, we can't make any conjectures 
as to the results, because it is impossible to guess how such machines might behave. We only 
point out that the fate of the human race would be at the mercy of the machines. It might be 
argued that the human race would never be foolish enough to hand over all the power to the 
machines. But we are suggesting neither that the human race would voluntarily turn power 
over to the machines nor that the machines would willfully seize power. What we do suggest 
is that the human race might easily permit itself to drift into a position of such dependence on 
the machines that it would have no practical choice but to accept all of the machines' 
decisions. As society and the problems that face it become more and more complex and 
machines become more and more intelligent, people will let machines make more of their 
decisions for them, simply because machine-made decisions will bring better results than 
man-made ones. Eventually a stage may be reached at which the decisions necessary to keep 
the system running will be so complex that human beings will be incapable of making them 
intelligently. At that stage the machines will be in effective control. People won't be able to 
just turn the machines off, because they will be so dependent on them that turning them off 


would amount to suicide. 


On the other hand it is possible that human control over the machines may be retained. In 
that case the average man may have control over certain private machines of his own, such as 
his car or his personal computer, but control over large systems of machines will be in the 
hands of a tiny elite—just as it is today, but with two differences. Due to improved techniques 
the elite will have greater control over the masses; and because human work will no longer be 
necessary the masses will be superfluous, a useless burden on the system. If the elite is 
ruthless they may simply decide to exterminate the mass of humanity. If they are humane 
they may use propaganda or other psychological or biological techniques to reduce the birth 
rate until the mass of humanity becomes extinct, leaving the world to the elite. Or, if the elite 
consists of soft-hearted liberals, they may decide to play the role of good shepherds to the rest 


of the human race. They will see to it that everyone's physical needs are satisfied, that all 
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children are raised under psychologically hygienic conditions, that everyone has a wholesome 
hobby to keep him busy, and that anyone who may become dissatisfied undergoes 
"treatment" to cure his "problem." Of course, life will be so purposeless that people will have 
to be biologically or psychologically engineered either to remove their need for the power 
process or make them "sublimate" their drive for power into some harmless hobby. These 
engineered human beings may be happy in such a society, but they will most certainly not be 


free. They will have been reduced to the status of domestic animals. 


In the book, you don't discover until you turn the page that the author of this passage is 
Theodore Kaczynski—the Unabomber. I am no apologist for Kaczynski. His bombs killed 
three people during a 17-year terror campaign and wounded many others. One of his bombs 
gravely injured my friend David Gelernter, one of the most brilliant and visionary computer 
scientists of our time. Like many of my colleagues, I felt that I could easily have been the 


Unabomber's next target. 


Kaczynski's actions were murderous and, in my view, criminally insane. He is clearly a 
Luddite, but simply saying this does not dismiss his argument; as difficult as it is for me to 
acknowledge, I saw some merit in the reasoning in this single passage. I felt compelled to 


confront it. 


Kaczynski's dystopian vision describes unintended consequences, a well-known problem 
with the design and use of technology, and one that is clearly related to Murphy's law 
—"Anything that can go wrong, will." (Actually, this is Finagle's law, which in itself shows that 
Finagle was right.) Our overuse of antibiotics has led to what may be the biggest such problem 
so far: the emergence of antibiotic-resistant and much more dangerous bacteria. Similar 
things happened when attempts to eliminate malarial mosquitoes using DDT caused them to 


acquire DDT resistance; malarial parasites likewise acquired multi-drug-resistant genes.* 


The cause of many such surprises seems clear: The systems involved are complex, involving 
interaction among and feedback between many parts. Any changes to such a system will 
cascade in ways that are difficult to predict; this is especially true when human actions are 


involved. 
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I started showing friends the Kaczynski quote from The Age of Spiritual Machines; I would 
hand them Kurzweil's book, let them read the quote, and then watch their reaction as they 
discovered who had written it. At around the same time, I found Hans Moravec's book Robot: 
Mere Machine to Transcendent Mind. Moravec is one of the leaders in robotics research, and 
was a founder of the world's largest robotics research program, at Carnegie Mellon University. 
Robot gave me more material to try out on my friends—material surprisingly supportive of 


Kaczynski's argument. For example: 
The Short Run (Early 2000s) 


Biological species almost never survive encounters with superior competitors. Ten million 
years ago, South and North America were separated by a sunken Panama isthmus. South 
America, like Australia today, was populated by marsupial mammals, including pouched 
equivalents of rats, deers, and tigers. When the isthmus connecting North and South America 
rose, it took only a few thousand years for the northern placental species, with slightly more 
effective metabolisms and reproductive and nervous systems, to displace and eliminate 


almost all the southern marsupials. 


In a completely free marketplace, superior robots would surely affect humans as North 
American placentals affected South American marsupials (and as humans have affected 
countless species). Robotic industries would compete vigorously among themselves for 
matter, energy, and space, incidentally driving their price beyond human reach. Unable to 


afford the necessities of life, biological humans would be squeezed out of existence. 


There is probably some breathing room, because we do not live in a completely free 
marketplace. Government coerces nonmarket behavior, especially by collecting taxes. 
Judiciously applied, governmental coercion could support human populations in high style on 


the fruits of robot labor, perhaps for a long while. 


A textbook dystopia—and Moravec is just getting wound up. He goes on to discuss how our 
main job in the 21st century will be "ensuring continued cooperation from the robot 
industries" by passing laws decreeing that they be "nice," and to describe how seriously 


dangerous a human can be "once transformed into an unbounded superintelligent robot." 
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Moravec's view is that the robots will eventually succeed us—that humans clearly face 


extinction. 


I decided it was time to talk to my friend Danny Hillis. Danny became famous as the 
cofounder of Thinking Machines Corporation, which built a very powerful parallel 
supercomputer. Despite my current job title of Chief Scientist at Sun Microsystems, I am more 
a computer architect than a scientist, and I respect Danny's knowledge of the information and 
physical sciences more than that of any other single person I know. Danny is also a highly 
regarded futurist who thinks long-term—four years ago he started the Long Now Foundation, 
which is building a clock designed to last 10,000 years, in an attempt to draw attention to the 


pitifully short attention span of our society. (See "Test of Time," Wired 8.03, page 78.) 


So I flew to Los Angeles for the express purpose of having dinner with Danny and his wife, 
Pati. I went through my now-familiar routine, trotting out the ideas and passages that I found 
so disturbing. Danny's answer—directed specifically at Kurzweil's scenario of humans 
merging with robots—came swiftly, and quite surprised me. He said, simply, that the changes 


would come gradually, and that we would get used to them. 


But I guess I wasn't totally surprised. I had seen a quote from Danny in Kurzweil's book in 
which he said, "I'm as fond of my body as anyone, but if I can be 200 with a body of silicon, 
I'll take it." It seemed that he was at peace with this process and its attendant risks, while I 


was not. 


While talking and thinking about Kurzweil, Kaczynski, and Moravec, I suddenly remembered 
a novel J had read almost 20 years ago -The White Plague, by Frank Herbert—in which a 
molecular biologist is driven insane by the senseless murder of his family. To seek revenge he 
constructs and disseminates a new and highly contagious plague that kills widely but 
selectively. (We're lucky Kaczynski was a mathematician, not a molecular biologist.) I was 
also reminded of the Borg ofStar Trek, a hive of partly biological, partly robotic creatures with 
a strong destructive streak. Borg-like disasters are a staple of science fiction, so why hadn't I 
been more concerned about such robotic dystopias earlier? Why weren't other people more 


concerned about these nightmarish scenarios? 
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Part of the answer certainly lies in our attitude toward the new—in our bias toward instant 
familiarity and unquestioning acceptance. Accustomed to living with almost routine scientific 
breakthroughs, we have yet to come to terms with the fact that the most compelling 21st- 
century technologies—robotics, genetic engineering, and nanotechnology—pose a different 
threat than the technologies that have come before. Specifically, robots, engineered 
organisms, and nanobots share a dangerous amplifying factor: They can self-replicate. A 


bomb is blown up only once—but one bot can become many, and quickly get out of control. 


Much of my work over the past 25 years has been on computer networking, where the 
sending and receiving of messages creates the opportunity for out-of-control replication. But 
while replication in a computer or a computer network can be a nuisance, at worst it disables 
a machine or takes down a network or network service. Uncontrolled self-replication in these 
newer technologies runs a much greater risk: a risk of substantial damage in the physical 


world. 


Each of these technologies also offers untold promise: The vision of near immortality that 
Kurzweil sees in his robot dreams drives us forward; genetic engineering may soon provide 
treatments, if not outright cures, for most diseases; and nanotechnology and nanomedicine 
can address yet more ills. Together they could significantly extend our average life span and 
improve the quality of our lives. Yet, with each of these technologies, a sequence of small, 
individually sensible advances leads to an accumulation of great power and, concomitantly, 


great danger. 


What was different in the 20th century? Certainly, the technologies underlying the weapons 
of mass destruction (WMD)—nuclear, biological, and chemical (NBC)—were powerful, and the 
weapons an enormous threat. But building nuclear weapons required, at least for a time, 
access to both rare—indeed, effectively unavailable—raw materials and highly protected 
information; biological and chemical weapons programs also tended to require large-scale 


activities. 


The 21st-century technologies—genetics, nanotechnology, and robotics (GNR)—are so 
powerful that they can spawn whole new classes of accidents and abuses. Most dangerously, 


for the first time, these accidents and abuses are widely within the reach of individuals or 
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small groups. They will not require large facilities or rare raw materials. Knowledge alone will 


enable the use of them. 


Thus we have the possibility not just of weapons of mass destruction but of knowledge- 
enabled mass destruction (KMD), this destructiveness hugely amplified by the power of self- 


replication. 


I think it is no exaggeration to say we are on the cusp of the further perfection of extreme evil, 
an evil whose possibility spreads well beyond that which weapons of mass destruction 
bequeathed to the nation-states, on to a surprising and terrible empowerment of extreme 


individuals. 


Nothing about the way I got involved with computers suggested to me that I was going to be 


facing these kinds of issues. 


My life has been driven by a deep need to ask questions and find answers. When I was 3, I 
was already reading, so my father took me to the elementary school, where I sat on the 
principal's lap and read him a story. I started school early, later skipped a grade, and escaped 
into books—I was incredibly motivated to learn. I asked lots of questions, often driving adults 


to distraction. 


As a teenager I was very interested in science and technology. I wanted to be a ham radio 
operator but didn't have the money to buy the equipment. Ham radio was the Internet of its 
time: very addictive, and quite solitary. Money issues aside, my mother put her foot down—I 


was not to be a ham; I was antisocial enough already. 


I may not have had many close friends, but I was awash in ideas. By high school, I had 
discovered the great science fiction writers. I remember especially Heinlein's Have Spacesuit 
Will Travel and Asimov's I, Robot, with its Three Laws of Robotics. I was enchanted by the 
descriptions of space travel, and wanted to have a telescope to look at the stars; since I had no 
money to buy or make one, I checked books on telescope-making out of the library and read 


about making them instead. I soared in my imagination. 


Thursday nights my parents went bowling, and we kids stayed home alone. It was the night of 


Gene Roddenberry's original Star Trek, and the program made a big impression on me. I came 
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to accept its notion that humans had a future in space, Western-style, with big heroes and 
adventures. Roddenberry's vision of the centuries to come was one with strong moral values, 
embodied in codes like the Prime Directive: to not interfere in the development of less 
technologically advanced civilizations. This had an incredible appeal to me; ethical humans, 


not robots, dominated this future, and I took Roddenberry's dream as part of my own. 


I excelled in mathematics in high school, and when I went to the University of Michigan as an 
undergraduate engineering student I took the advanced curriculum of the mathematics 
majors. Solving math problems was an exciting challenge, but when I discovered computers I 
found something much more interesting: a machine into which you could put a program that 
attempted to solve a problem, after which the machine quickly checked the solution. The 
computer had a clear notion of correct and incorrect, true and false. Were my ideas correct? 


The machine could tell me. This was very seductive. 


I was lucky enough to get a job programming early supercomputers and discovered the 
amazing power of large machines to numerically simulate advanced designs. When I went to 
graduate school at UC Berkeley in the mid-1970s, I started staying up late, often all night, 
inventing new worlds inside the machines. Solving problems. Writing the code that argued so 


strongly to be written. 


In The Agony and the Ecstasy, Irving Stone's biographical novel of Michelangelo, Stone 
described vividly how Michelangelo released the statues from the stone, "breaking the marble 
spell,” carving from the images in his mind.‘ In my most ecstatic moments, the software in the 
computer emerged in the same way. Once I had imagined it in my mind I felt that it was 
already there in the machine, waiting to be released. Staying up all night seemed a small price 


to pay to free it—to give the ideas concrete form. 


After a few years at Berkeley I started to send out some of the software I had written—an 
instructional Pascal system, Unix utilities, and a text editor called vi (which is still, to my 
surprise, widely used more than 20 years later)—to others who had similar small PDP-11 and 
VAX minicomputers. These adventures in software eventually turned into the Berkeley version 
of the Unix operating system, which became a personal "success disaster"—so many people 
wanted it that I never finished my PhD. Instead I got a job working for Darpa putting Berkeley 


Unix on the Internet and fixing it to be reliable and to run large research applications well. 
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This was all great fun and very rewarding. And, frankly, I saw no robots here, or anywhere 


near. 


Still, by the early 1980s, I was drowning. The Unix releases were very successful, and my little 
project of one soon had money and some staff, but the problem at Berkeley was always office 
space rather than money—there wasn't room for the help the project needed, so when the 
other founders of Sun Microsystems showed up I jumped at the chance to join them. At Sun, 
the long hours continued into the early days of workstations and personal computers, and I 
have enjoyed participating in the creation of advanced microprocessor technologies and 


Internet technologies such as Java and Jini. 


From all this, I trust it is clear that Iam not a Luddite. I have always, rather, had a strong belief 
in the value of the scientific search for truth and in the ability of great engineering to bring 
material progress. The Industrial Revolution has immeasurably improved everyone's life over 
the last couple hundred years, and I always expected my career to involve the building of 


worthwhile solutions to real problems, one problem at a time. 


I have not been disappointed. My work has had more impact than I had ever hoped for and 
has been more widely used than I could have reasonably expected. I have spent the last 20 
years still trying to figure out how to make computers as reliable as I want them to be (they 
are not nearly there yet) and how to make them simple to use (a goal that has met with even 
less relative success). Despite some progress, the problems that remain seem even more 


daunting. 


But while I was aware of the moral dilemmas surrounding technology's consequences in 
fields like weapons research, I did not expect that I would confront such issues in my own 


field, or at least not so soon. 


Perhaps it is always hard to see the bigger impact while you are in the vortex of a change. 
Failing to understand the consequences of our inventions while we are in the rapture of 
discovery and innovation seems to be a common fault of scientists and technologists; we have 
long been driven by the overarching desire to know that is the nature of science's quest, not 
stopping to notice that the progress to newer and more powerful technologies can take ona 


life of its own. 


https://www.wired.com/2000/04/joy-2/ 11/33 


6/19/2021 Why the Future Doesn't Need Us | WIRED 
I have long realized that the big advances in information technology come not from the work 
of computer scientists, computer architects, or electrical engineers, but from that of physical 
scientists. The physicists Stephen Wolfram and Brosl Hasslacher introduced me, in the early 
1980s, to chaos theory and nonlinear systems. In the 1990s, I learned about complex systems 
from conversations with Danny Hillis, the biologist Stuart Kauffman, the Nobel-laureate 
physicist Murray Gell-Mann, and others. Most recently, Hasslacher and the electrical engineer 
and device physicist Mark Reed have been giving me insight into the incredible possibilities of 


molecular electronics. 


In my own work, as codesigner of three microprocessor architectures—SPARC, picoJava, and 
MAJC—and as the designer of several implementations thereof, I've been afforded a deep and 
firsthand acquaintance with Moore's law. For decades, Moore's law has correctly predicted the 
exponential rate of improvement of semiconductor technology. Until last year I believed that 
the rate of advances predicted by Moore's law might continue only until roughly 2010, when 
some physical limits would begin to be reached. It was not obvious to me that a new 


technology would arrive in time to keep performance advancing smoothly. 


But because of the recent rapid and radical progress in molecular electronics—where 
individual atoms and molecules replace lithographically drawn transistors—and related 
nanoscale technologies, we should be able to meet or exceed the Moore's law rate of progress 
for another 30 years. By 2030, we are likely to be able to build machines, in quantity, a 
million times as powerful as the personal computers of today—sufficient to implement the 


dreams of Kurzweil and Moravec. 


As this enormous computing power is combined with the manipulative advances of the 
physical sciences and the new, deep understandings in genetics, enormous transformative 
power is being unleashed. These combinations open up the opportunity to completely 
redesign the world, for better or worse: The replicating and evolving processes that have been 


confined to the natural world are about to become realms of human endeavor. 


In designing software and microprocessors, I have never had the feeling that I was designing 
an intelligent machine. The software and hardware is so fragile and the capabilities of the 
machine to "think" so clearly absent that, even as a possibility, this has always seemed very 


far in the future. 
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But now, with the prospect of human-level computing power in about 30 years, a new idea 
suggests itself: that I may be working to create tools which will enable the construction of the 
technology that may replace our species. How do I feel about this? Very uncomfortable. 
Having struggled my entire career to build reliable software systems, it seems to me more 
than likely that this future will not work out as well as some people may imagine. My personal 


experience suggests we tend to overestimate our design abilities. 


Given the incredible power of these new technologies, shouldn't we be asking how we can 
best coexist with them? And if our own extinction is a likely, or even possible, outcome of our 


technological development, shouldn't we proceed with great caution? 


The dream of robotics is, first, that intelligent machines can do our work for us, allowing us 
lives of leisure, restoring us to Eden. Yet in his history of such ideas, Darwin Among the 
Machines, George Dyson warns: "In the game of life and evolution there are three players at 
the table: human beings, nature, and machines. I am firmly on the side of nature. But nature, I 
suspect, is on the side of the machines." As we have seen, Moravec agrees, believing we may 


well not survive the encounter with the superior robot species. 


How soon could such an intelligent robot be built? The coming advances in computing power 
seem to make it possible by 2030. And once an intelligent robot exists, it is only a small step 


to a robot species—to an intelligent robot that can make evolved copies of itself. 


A second dream of robotics is that we will gradually replace ourselves with our robotic 
technology, achieving near immortality by downloading our consciousnesses; it is this process 
that Danny Hillis thinks we will gradually get used to and that Ray Kurzweil elegantly details 
in The Age of Spiritual Machines. (We are beginning to see intimations of this in the 
implantation of computer devices into the human body, as illustrated on the cover of Wired 
8.02.) 


But if we are downloaded into our technology, what are the chances that we will thereafter be 
ourselves or even human? It seems to me far more likely that a robotic existence would not be 
like a human one in any sense that we understand, that the robots would in no sense be our 


children, that on this path our humanity may well be lost. 


https://www.wired.com/2000/04/joy-2/ 13/33 


6/19/2021 Why the Future Doesn't Need Us | WIRED 
Genetic engineering promises to revolutionize agriculture by increasing crop yields while 
reducing the use of pesticides; to create tens of thousands of novel species of bacteria, plants, 
viruses, and animals; to replace reproduction, or supplement it, with cloning; to create cures 
for many diseases, increasing our life span and our quality of life; and much, much more. We 
now know with certainty that these profound changes in the biological sciences are imminent 


and will challenge all our notions of what life is. 


Technologies such as human cloning have in particular raised our awareness of the profound 
ethical and moral issues we face. If, for example, we were to reengineer ourselves into several 
separate and unequal species using the power of genetic engineering, then we would threaten 


the notion of equality that is the very cornerstone of our democracy. 


Given the incredible power of genetic engineering, it's no surprise that there are significant 
safety issues in its use. My friend Amory Lovins recently cowrote, along with Hunter Lovins, 
an editorial that provides an ecological view of some of these dangers. Among their concerns: 
that "the new botany aligns the development of plants with their economic, not evolutionary, 
success." (See "A Tale of Two Botanies," page 247.) 

Amory's long career has been focused on energy and resource efficiency by taking a whole- 
system view of human-made systems; such a whole-system view often finds simple, smart 


solutions to otherwise seemingly difficult problems, and is usefully applied here as well. 


After reading the Lovins' editorial, I saw an op-ed by Gregg Easterbrook in The New York 
Times (November 19, 1999) about genetically engineered crops, under the headline: "Food for 


the Future: Someday, rice will have built-in vitamin A. Unless the Luddites win." 


Are Amory and Hunter Lovins Luddites? Certainly not. I believe we all would agree that 
golden rice, with its built-in vitamin A, is probably a good thing, if developed with proper care 


and respect for the likely dangers in moving genes across species boundaries. 


Awareness of the dangers inherent in genetic engineering is beginning to grow, as reflected in 
the Lovins' editorial. The general public is aware of, and uneasy about, genetically modified 
foods, and seems to be rejecting the notion that such foods should be permitted to be 


unlabeled. 
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But genetic engineering technology is already very far along. As the Lovins note, the USDA has 
already approved about 50 genetically engineered crops for unlimited release; more than half 
of the world's soybeans and a third of its corn now contain genes spliced in from other forms 


of life. 


While there are many important issues here, my own major concern with genetic engineering 
is narrower: that it gives the power—whether militarily, accidentally, or in a deliberate 


terrorist act—to create a White Plague. 


The many wonders of nanotechnology were first imagined by the Nobel-laureate physicist 
Richard Feynman in a speech he gave in 1959, subsequently published under the title "There's 
Plenty of Room at the Bottom.” The book that made a big impression on me, in the mid-'80s, 
was Eric Drexler's Engines of Creation, in which he described beautifully how manipulation of 
matter at the atomic level could create a utopian future of abundance, where just about 
everything could be made cheaply, and almost any imaginable disease or physical problem 


could be solved using nanotechnology and artificial intelligences. 


A subsequent book, Unbounding the Future: The Nanotechnology Revolution, which Drexler 
cowrote, imagines some of the changes that might take place in a world where we had 
molecular-level "assemblers." Assemblers could make possible incredibly low-cost solar 
power, cures for cancer and the common cold by augmentation of the human immune 
system, essentially complete cleanup of the environment, incredibly inexpensive pocket 
supercomputers—in fact, any product would be manufacturable by assemblers at a cost no 
greater than that of wood—spaceflight more accessible than transoceanic travel today, and 


restoration of extinct species. 


I remember feeling good about nanotechnology after reading Engines of Creation. Asa 
technologist, it gave me a sense of calm—that is, nanotechnology showed us that incredible 
progress was possible, and indeed perhaps inevitable. If nanotechnology was our future, then 
I didn't feel pressed to solve so many problems in the present. I would get to Drexler's utopian 
future in due time; I might as well enjoy life more in the here and now. It didn't make sense, 


given his vision, to stay up all night, all the time. 
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Drexler's vision also led to a lot of good fun. I would occasionally get to describe the wonders 
of nanotechnology to others who had not heard of it. After teasing them with all the things 
Drexler described I would give a homework assignment of my own: "Use nanotechnology to 


create a vampire; for extra credit create an antidote." 


With these wonders came clear dangers, of which I was acutely aware. As I said ata 
nanotechnology conference in 1989, "We can't simply do our science and not worry about 
these ethical issues" But my subsequent conversations with physicists convinced me that 
nanotechnology might not even work—or, at least, it wouldn't work anytime soon. Shortly 
thereafter I moved to Colorado, to a skunk works I had set up, and the focus of my work 


shifted to software for the Internet, specifically on ideas that became Java and Jini. 


Then, last summer, Brosl Hasslacher told me that nanoscale molecular electronics was now 
practical. This was new news, at least to me, and I think to many people—and it radically 
changed my opinion about nanotechnology. It sent me back to Engines of Creation. Rereading 
Drexler's work after more than 10 years, I was dismayed to realize how little I had 
remembered of its lengthy section called "Dangers and Hopes,” including a discussion of how 
nanotechnologies can become "engines of destruction.” Indeed, in my rereading of this 
cautionary material today, I am struck by how naive some of Drexler's safeguard proposals 
seem, and how much greater I judge the dangers to be now than even he seemed to then. 
(Having anticipated and described many technical and political problems with 
nanotechnology, Drexler started the Foresight Institute in the late 1980s "to help prepare 


society for anticipated advanced technologies"—most important, nanotechnology.) 


The enabling breakthrough to assemblers seems quite likely within the next 20 years. 
Molecular electronics—the new subfield of nanotechnology where individual molecules are 
circuit elements—should mature quickly and become enormously lucrative within this 


decade, causing a large incremental investment in all nanotechnologies. 


Unfortunately, as with nuclear technology, it is far easier to create destructive uses for 
nanotechnology than constructive ones. Nanotechnology has clear military and terrorist uses, 
and you need not be suicidal to release a massively destructive nanotechnological device— 
such devices can be built to be selectively destructive, affecting, for example, only a certain 


geographical area or a group of people who are genetically distinct. 
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An immediate consequence of the Faustian bargain in obtaining the great power of 
nanotechnology is that we run a grave risk—the risk that we might destroy the biosphere on 


which all life depends. 
As Drexler explained: 


"Plants" with "leaves" no more efficient than today's solar cells could out-compete real plants, 
crowding the biosphere with an inedible foliage. Tough omnivorous "bacteria" could out- 
compete real bacteria: They could spread like blowing pollen, replicate swiftly, and reduce the 
biosphere to dust in a matter of days. Dangerous replicators could easily be too tough, small, 
and rapidly spreading to stop—at least if we make no preparation. We have trouble enough 


controlling viruses and fruit flies. 


Among the cognoscenti of nanotechnology, this threat has become known as the "gray goo 
problem." Though masses of uncontrolled replicators need not be gray or gooey, the term 
"gray goo" emphasizes that replicators able to obliterate life might be less inspiring than a 
single species of crabgrass. They might be superior in an evolutionary sense, but this need not 


make them valuable. 


The gray goo threat makes one thing perfectly clear: We cannot afford certain kinds of 


accidents with replicating assemblers. 


Gray goo would surely be a depressing ending to our human adventure on Earth, far worse 


than mere fire or ice, and one that could stem from a simple laboratory accident.® Oops. 


It is most of all the power of destructive self-replication in genetics, nanotechnology, and 
robotics (GNR) that should give us pause. Self-replication is the modus operandi of genetic 
engineering, which uses the machinery of the cell to replicate its designs, and the prime 
danger underlying gray goo in nanotechnology. Stories of run-amok robots like the Borg, 
replicating or mutating to escape from the ethical constraints imposed on them by their 
creators, are well established in our science fiction books and movies. It is even possible that 
self-replication may be more fundamental than we thought, and hence harder—or even 
impossible—to control. A recent article by Stuart Kauffman in Nature titled "Self-Replication: 
Even Peptides Do It" discusses the discovery that a 32-amino-acid peptide can "autocatalyse 
its own synthesis." We don't know how widespread this ability is, but Kauffman notes that it 
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may hint at "a route to self-reproducing molecular systems on a basis far wider than Watson- 


Crick base-pairing.” 


In truth, we have had in hand for years clear warnings of the dangers inherent in widespread 
knowledge of GNR technologies—of the possibility of knowledge alone enabling mass 
destruction. But these warnings haven't been widely publicized; the public discussions have 


been clearly inadequate. There is no profit in publicizing the dangers. 


The nuclear, biological, and chemical (NBC) technologies used in 20th-century weapons of 
mass destruction were and are largely military, developed in government laboratories. In 
sharp contrast, the 21st-century GNR technologies have clear commercial uses and are being 
developed almost exclusively by corporate enterprises. In this age of triumphant 
commercialism, technology—with science as its handmaiden—is delivering a series of almost 
magical inventions that are the most phenomenally lucrative ever seen. We are aggressively 
pursuing the promises of these new technologies within the now-unchallenged system of 


global capitalism and its manifold financial incentives and competitive pressures. 


This is the first moment in the history of our planet when any species, by its own voluntary 


actions, has become a danger to itself—as well as to vast numbers of others. 


It might be a familiar progression, transpiring on many worlds—a planet, newly formed, 
placidly revolves around its star; life slowly forms; a kaleidoscopic procession of creatures 
evolves; intelligence emerges which, at least up to a point, confers enormous survival value; 
and then technology is invented. It dawns on them that there are such things as laws of 
Nature, that these laws can be revealed by experiment, and that knowledge of these laws can 
be made both to save and to take lives, both on unprecedented scales. Science, they 
recognize, grants immense powers. In a flash, they create world-altering contrivances. Some 
planetary civilizations see their way through, place limits on what may and what must not be 


done, and safely pass through the time of perils. Others, not so lucky or so prudent, perish. 


That is Carl Sagan, writing in 1994, in Pale Blue Dot, a book describing his vision of the human 
future in space. I am only now realizing how deep his insight was, and how sorely I miss, and 


will miss, his voice. For all its eloquence, Sagan's contribution was not least that of simple 
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common sense—an attribute that, along with humility, many of the leading advocates of the 


21st-century technologies seem to lack. 


I remember from my childhood that my grandmother was strongly against the overuse of 
antibiotics. She had worked since before the first World War as a nurse and had a 
commonsense attitude that taking antibiotics, unless they were absolutely necessary, was bad 


for you. 


It is not that she was an enemy of progress. She saw much progress in an almost 70-year 
nursing career; my grandfather, a diabetic, benefited greatly from the improved treatments 
that became available in his lifetime. But she, like many levelheaded people, would probably 
think it greatly arrogant for us, now, to be designing a robotic "replacement species," when we 
obviously have so much trouble making relatively simple things work, and so much trouble 


managing—or even understanding—ourselves. 


I realize now that she had an awareness of the nature of the order of life, and of the necessity 
of living with and respecting that order. With this respect comes a necessary humility that we, 
with our early-21st-century chutzpah, lack at our peril. The commonsense view, grounded in 
this respect, is often right, in advance of the scientific evidence. The clear fragility and 
inefficiencies of the human-made systems we have built should give us all pause; the fragility 


of the systems I have worked on certainly humbles me. 


We should have learned a lesson from the making of the first atomic bomb and the resulting 


arms race. We didn't do well then, and the parallels to our current situation are troubling. 


The effort to build the first atomic bomb was led by the brilliant physicist J. Robert 
Oppenheimer. Oppenheimer was not naturally interested in politics but became painfully 
aware of what he perceived as the grave threat to Western civilization from the Third Reich, a 
threat surely grave because of the possibility that Hitler might obtain nuclear weapons. 
Energized by this concern, he brought his strong intellect, passion for physics, and charismatic 
leadership skills to Los Alamos and led a rapid and successful effort by an incredible 


collection of great minds to quickly invent the bomb. 


What is striking is how this effort continued so naturally after the initial impetus was 
removed. In a meeting shortly after V-E Day with some physicists who felt that perhaps the 


https://www.wired.com/2000/04/joy-2/ 19/33 


6/19/2021 Why the Future Doesn't Need Us | WIRED 
effort should stop, Oppenheimer argued to continue. His stated reason seems a bit strange: 
not because of the fear of large casualties from an invasion of Japan, but because the United 
Nations, which was soon to be formed, should have foreknowledge of atomic weapons. A 
more likely reason the project continued is the momentum that had built up—the first atomic 


test, Trinity, was nearly at hand. 


We know that in preparing this first atomic test the physicists proceeded despite a large 
number of possible dangers. They were initially worried, based on a calculation by Edward 
Teller, that an atomic explosion might set fire to the atmosphere. A revised calculation 
reduced the danger of destroying the world to a three-in-a-million chance. (Teller says he 
was later able to dismiss the prospect of atmospheric ignition entirely.) Oppenheimer, though, 
was sufficiently concerned about the result of Trinity that he arranged for a possible 
evacuation of the southwest part of the state of New Mexico. And, of course, there was the 


clear danger of starting a nuclear arms race. 


Within a month of that first, successful test, two atomic bombs destroyed Hiroshima and 
Nagasaki. Some scientists had suggested that the bomb simply be demonstrated, rather than 
dropped on Japanese cities—saying that this would greatly improve the chances for arms 
control after the war—but to no avail. With the tragedy of Pearl Harbor still fresh in 
Americans’ minds, it would have been very difficult for President Truman to order a 
demonstration of the weapons rather than use them as he did—the desire to quickly end the 
war and save the lives that would have been lost in any invasion of Japan was very strong. Yet 
the overriding truth was probably very simple: As the physicist Freeman Dyson later said, 
"The reason that it was dropped was just that nobody had the courage or the foresight to say 


no. 


It's important to realize how shocked the physicists were in the aftermath of the bombing of 
Hiroshima, on August 6, 1945. They describe a series of waves of emotion: first, a sense of 
fulfillment that the bomb worked, then horror at all the people that had been killed, and then 
a convincing feeling that on no account should another bomb be dropped. Yet of course 


another bomb was dropped, on Nagasaki, only three days after the bombing of Hiroshima. 


In November 1945, three months after the atomic bombings, Oppenheimer stood firmly 


behind the scientific attitude, saying, "It is not possible to be a scientist unless you believe that 
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the knowledge of the world, and the power which this gives, is a thing which is of intrinsic 
value to humanity, and that you are using it to help in the spread of knowledge and are willing 


to take the consequences." 


Oppenheimer went on to work, with others, on the Acheson-Lilienthal report, which, as 
Richard Rhodes says in his recent book Visions of Technology, "found a way to prevent a 
clandestine nuclear arms race without resorting to armed world government"; their 
suggestion was a form of relinquishment of nuclear weapons work by nation-states to an 


international agency. 


This proposal led to the Baruch Plan, which was submitted to the United Nations in June 1946 
but never adopted (perhaps because, as Rhodes suggests, Bernard Baruch had "insisted on 
burdening the plan with conventional sanctions," thereby inevitably dooming it, even though 
it would "almost certainly have been rejected by Stalinist Russia anyway"). Other efforts to 
promote sensible steps toward internationalizing nuclear power to prevent an arms race ran 
afoul either of US politics and internal distrust, or distrust by the Soviets. The opportunity to 


avoid the arms race was lost, and very quickly. 


Two years later, in 1948, Oppenheimer seemed to have reached another stage in his thinking, 
saying, "In some sort of crude sense which no vulgarity, no humor, no overstatement can 


quite extinguish, the physicists have known sin; and this is a knowledge they cannot lose." 


In 1949, the Soviets exploded an atom bomb. By 1955, both the US and the Soviet Union had 


tested hydrogen bombs suitable for delivery by aircraft. And so the nuclear arms race began. 


Nearly 20 years ago, in the documentary The Day After Trinity, Freeman Dyson summarized 


the scientific attitudes that brought us to the nuclear precipice: 


"I have felt it myself. The glitter of nuclear weapons. It is irresistible if you come to them asa 
scientist. To feel it's there in your hands, to release this energy that fuels the stars, to let it do 
your bidding. To perform these miracles, to lift a million tons of rock into the sky. It is 
something that gives people an illusion of illimitable power, and it is, in some ways, 
responsible for all our troubles—this, what you might call technical arrogance, that overcomes 


people when they see what they can do with their minds."8 
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Now, as then, we are creators of new technologies and stars of the imagined future, driven— 
this time by great financial rewards and global competition—despite the clear dangers, hardly 
evaluating what it may be like to try to live in a world that is the realistic outcome of what we 


are creating and imagining. 


In 1947, The Bulletin of the Atomic Scientists began putting a Doomsday Clock on its cover. 
For more than 50 years, it has shown an estimate of the relative nuclear danger we have 
faced, reflecting the changing international conditions. The hands on the clock have moved 15 
times and today, standing at nine minutes to midnight, reflect continuing and real danger 
from nuclear weapons. The recent addition of India and Pakistan to the list of nuclear powers 
has increased the threat of failure of the nonproliferation goal, and this danger was reflected 


by moving the hands closer to midnight in 1998. 


In our time, how much danger do we face, not just from nuclear weapons, but from all of 


these technologies? How high are the extinction risks? 


The philosopher John Leslie has studied this question and concluded that the risk of human 
extinction is at least 30 percent, while Ray Kurzweil believes we have "a better than even 
chance of making it through," with the caveat that he has "always been accused of being an 
optimist"? Not only are these estimates not encouraging, but they do not include the 


probability of many horrid outcomes that lie short of extinction. 


Faced with such assessments, some serious people are already suggesting that we simply 
move beyond Earth as quickly as possible. We would colonize the galaxy using von Neumann 
probes, which hop from star system to star system, replicating as they go. This step will almost 
certainly be necessary 5 billion years from now (or sooner if our solar system is disastrously 
impacted by the impending collision of our galaxy with the Andromeda galaxy within the next 
3 billion years), but if we take Kurzweil and Moravec at their word it might be necessary by 


the middle of this century. 


What are the moral implications here? If we must move beyond Earth this quickly in order for 
the species to survive, who accepts the responsibility for the fate of those (most of us, after all) 


who are left behind? And even if we scatter to the stars, isn't it likely that we may take our 


https://www.wired.com/2000/04/joy-2/ 22/33 


6/19/2021 Why the Future Doesn't Need Us | WIRED 
problems with us or find, later, that they have followed us? The fate of our species on Earth 


and our fate in the galaxy seem inextricably linked. 


Another idea is to erect a series of shields to defend against each of the dangerous 
technologies. The Strategic Defense Initiative, proposed by the Reagan administration, was an 
attempt to design such a shield against the threat of a nuclear attack from the Soviet Union. 
But as Arthur C. Clarke, who was privy to discussions about the project, observed: "Though it 
might be possible, at vast expense, to construct local defense systems that would 'only' let 
through a few percent of ballistic missiles, the much touted idea of a national umbrella was 
nonsense. Luis Alvarez, perhaps the greatest experimental physicist of this century, remarked 


to me that the advocates of such schemes were 'very bright guys with no common sense." 


Clarke continued: "Looking into my often cloudy crystal ball, I suspect that a total defense 
might indeed be possible in a century or so. But the technology involved would produce, as a 
by-product, weapons so terrible that no one would bother with anything as primitive as 


ballistic missiles."!9 


In Engines of Creation, Eric Drexler proposed that we build an active nanotechnological 
shield—a form of immune system for the biosphere—to defend against dangerous replicators 
of all kinds that might escape from laboratories or otherwise be maliciously created. But the 
shield he proposed would itself be extremely dangerous—nothing could prevent it from 


developing autoimmune problems and attacking the biosphere itself.!! 


Similar difficulties apply to the construction of shields against robotics and genetic 
engineering. These technologies are too powerful to be shielded against in the time frame of 
interest; even if it were possible to implement defensive shields, the side effects of their 
development would be at least as dangerous as the technologies we are trying to protect 


against. 


These possibilities are all thus either undesirable or unachievable or both. The only realistic 
alternative I see is relinquishment: to limit development of the technologies that are too 


dangerous, by limiting our pursuit of certain kinds of knowledge. 


Yes, I know, knowledge is good, as is the search for new truths. We have been seeking 
knowledge since ancient times. Aristotle opened his Metaphysics with the simple statement: 
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"All men by nature desire to know." We have, as a bedrock value in our society, long agreed on 
the value of open access to information, and recognize the problems that arise with attempts 
to restrict access to and development of knowledge. In recent times, we have come to revere 


scientific knowledge. 


But despite the strong historical precedents, if open access to and unlimited development of 
knowledge henceforth puts us all in clear danger of extinction, then common sense demands 


that we reexamine even these basic, long-held beliefs. 


It was Nietzsche who warned us, at the end of the 19th century, not only that God is dead but 
that "faith in science, which after all exists undeniably, cannot owe its origin to a calculus of 
utility; it must have originated in spite of the fact that the disutility and dangerousness of the 
‘will to truth; of ‘truth at any price’ is proved to it constantly." It is this further danger that we 
now fully face—the consequences of our truth-seeking. The truth that science seeks can 


certainly be considered a dangerous substitute for God if it is likely to lead to our extinction. 


If we could agree, as a species, what we wanted, where we were headed, and why, then we 
would make our future much less dangerous—then we might understand what we can and 
should relinquish. Otherwise, we can easily imagine an arms race developing over GNR 
technologies, as it did with the NBC technologies in the 20th century. This is perhaps the 
greatest risk, for once such a race begins, it's very hard to end it. This time—unlike during the 
Manhattan Project—we aren't in a war, facing an implacable enemy that is threatening our 
civilization; we are driven, instead, by our habits, our desires, our economic system, and our 


competitive need to know. 


I believe that we all wish our course could be determined by our collective values, ethics, and 
morals. If we had gained more collective wisdom over the past few thousand years, thena 
dialogue to this end would be more practical, and the incredible powers we are about to 


unleash would not be nearly so troubling. 


One would think we might be driven to such a dialogue by our instinct for self-preservation. 
Individuals clearly have this desire, yet as a species our behavior seems to be not in our favor. 
In dealing with the nuclear threat, we often spoke dishonestly to ourselves and to each other, 


thereby greatly increasing the risks. Whether this was politically motivated, or because we 
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chose not to think ahead, or because when faced with such grave threats we acted irrationally 


out of fear, I do not know, but it does not bode well. 


The new Pandora's boxes of genetics, nanotechnology, and robotics are almost open, yet we 
seem hardly to have noticed. Ideas can't be put back in a box; unlike uranium or plutonium, 
they don't need to be mined and refined, and they can be freely copied. Once they are out, 
they are out. Churchill remarked, in a famous left-handed compliment, that the American 
people and their leaders "invariably do the right thing, after they have examined every other 
alternative." In this case, however, we must act more presciently, as to do the right thing only 


at last may be to lose the chance to do it at all. 


As Thoreau said, "We do not ride on the railroad; it rides upon us"; and this is what we must 
fight, in our time. The question is, indeed, Which is to be master? Will we survive our 


technologies? 


We are being propelled into this new century with no plan, no control, no brakes. Have we 
already gone too far down the path to alter course? I don't believe so, but we aren't trying yet, 
and the last chance to assert control—the fail-safe point—is rapidly approaching. We have our 
first pet robots, as well as commercially available genetic engineering techniques, and our 
nanoscale techniques are advancing rapidly. While the development of these technologies 
proceeds through a number of steps, it isn't necessarily the case—as happened in the 
Manhattan Project and the Trinity test—that the last step in proving a technology is large and 
hard. The breakthrough to wild self-replication in robotics, genetic engineering, or 
nanotechnology could come suddenly, reprising the surprise we felt when we learned of the 


cloning of a mammal. 


And yet I believe we do have a strong and solid basis for hope. Our attempts to deal with 
weapons of mass destruction in the last century provide a shining example of relinquishment 
for us to consider: the unilateral US abandonment, without preconditions, of the development 
of biological weapons. This relinquishment stemmed from the realization that while it would 
take an enormous effort to create these terrible weapons, they could from then on easily be 


duplicated and fall into the hands of rogue nations or terrorist groups. 
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The clear conclusion was that we would create additional threats to ourselves by pursuing 
these weapons, and that we would be more secure if we did not pursue them. We have 
embodied our relinquishment of biological and chemical weapons in the 1972 Biological 


Weapons Convention (BWC) and the 1993 Chemical Weapons Convention (CWC).!4 


As for the continuing sizable threat from nuclear weapons, which we have lived with now for 
more than 50 years, the US Senate's recent rejection of the Comprehensive Test Ban Treaty 
makes it clear relinquishing nuclear weapons will not be politically easy. But we have a 
unique opportunity, with the end of the Cold War, to avert a multipolar arms race. Building on 
the BWC and CWC relinquishments, successful abolition of nuclear weapons could help us 
build toward a habit of relinquishing dangerous technologies. (Actually, by getting rid of all 
but 100 nuclear weapons worldwide—roughly the total destructive power of World War II and 


a considerably easier task—we could eliminate this extinction threat. 


Verifying relinquishment will be a difficult problem, but not an unsolvable one. We are 
fortunate to have already done a lot of relevant work in the context of the BWC and other 
treaties. Our major task will be to apply this to technologies that are naturally much more 
commercial than military. The substantial need here is for transparency, as difficulty of 
verification is directly proportional to the difficulty of distinguishing relinquished from 


legitimate activities. 


I frankly believe that the situation in 1945 was simpler than the one we now face: The nuclear 
technologies were reasonably separable into commercial and military uses, and monitoring 
was aided by the nature of atomic tests and the ease with which radioactivity could be 
measured. Research on military applications could be performed at national laboratories such 


as Los Alamos, with the results kept secret as long as possible. 


The GNR technologies do not divide clearly into commercial and military uses; given their 
potential in the market, it's hard to imagine pursuing them only in national laboratories. With 
their widespread commercial pursuit, enforcing relinquishment will require a verification 
regime similar to that for biological weapons, but on an unprecedented scale. This, inevitably, 
will raise tensions between our individual privacy and desire for proprietary information, and 
the need for verification to protect us all. We will undoubtedly encounter strong resistance to 


this loss of privacy and freedom of action. 
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Verifying the relinquishment of certain GNR technologies will have to occur in cyberspace as 
well as at physical facilities. The critical issue will be to make the necessary transparency 
acceptable in a world of proprietary information, presumably by providing new forms of 


protection for intellectual property. 


Verifying compliance will also require that scientists and engineers adopt a strong code of 
ethical conduct, resembling the Hippocratic oath, and that they have the courage to 
whistleblow as necessary, even at high personal cost. This would answer the call—50 years 
after Hiroshima—by the Nobel laureate Hans Bethe, one of the most senior of the surviving 
members of the Manhattan Project, that all scientists "cease and desist from work creating, 
developing, improving, and manufacturing nuclear weapons and other weapons of potential 
mass destruction."!4 In the 21st century, this requires vigilance and personal responsibility by 
those who would work on both NBC and GNR technologies to avoid implementing weapons 


of mass destruction and knowledge-enabled mass destruction. 


Thoreau also said that we will be "rich in proportion to the number of things which we can 
afford to let alone." We each seek to be happy, but it would seem worthwhile to question 
whether we need to take such a high risk of total destruction to gain yet more knowledge and 
yet more things; common sense says that there is a limit to our material needs—and that 


certain knowledge is too dangerous and is best forgone. 


Neither should we pursue near immortality without considering the costs, without 
considering the commensurate increase in the risk of extinction. Immortality, while perhaps 


the original, is certainly not the only possible utopian dream. 


I recently had the good fortune to meet the distinguished author and scholar Jacques Attali, 
whose book Lignes d'horizons ( Millennium, in the English translation) helped inspire the Java 
and Jini approach to the coming age of pervasive computing, as previously described in this 
magazine. In his new book Fraternités, Attali describes how our dreams of utopia have 


changed over time: 


"At the dawn of societies, men saw their passage on Earth as nothing more than a labyrinth of 
pain, at the end of which stood a door leading, via their death, to the company of gods and to 


Eternity. With the Hebrews and then the Greeks, some men dared free themselves from 
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theological demands and dream of an ideal City where Liberty would flourish. Others, noting 
the evolution of the market society, understood that the liberty of some would entail the 


alienation of others, and they sought Equality." 


Jacques helped me understand how these three different utopian goals exist in tension in our 
society today. He goes on to describe a fourth utopia, Fraternity, whose foundation is altruism. 
Fraternity alone associates individual happiness with the happiness of others, affording the 


promise of self-sustainment. 


This crystallized for me my problem with Kurzweil's dream. A technological approach to 
Eternity—near immortality through robotics—may not be the most desirable utopia, and its 


pursuit brings clear dangers. Maybe we should rethink our utopian choices. 


Where can we look for a new ethical basis to set our course? I have found the ideas in the 
book Ethics for the New Millennium, by the Dalai Lama, to be very helpful. As is perhaps well 
known but little heeded, the Dalai Lama argues that the most important thing is for us to 
conduct our lives with love and compassion for others, and that our societies need to develop 
a stronger notion of universal responsibility and of our interdependency; he proposes a 
standard of positive ethical conduct for individuals and societies that seems consonant with 


Attali's Fraternity utopia. 


The Dalai Lama further argues that we must understand what it is that makes people happy, 
and acknowledge the strong evidence that neither material progress nor the pursuit of the 
power of knowledge is the key—that there are limits to what science and the scientific pursuit 


alone can do. 


Our Western notion of happiness seems to come from the Greeks, who defined it as "the 


exercise of vital powers along lines of excellence in a life affording them scope." 


Clearly, we need to find meaningful challenges and sufficient scope in our lives if we are to be 
happy in whatever is to come. But I believe we must find alternative outlets for our creative 
forces, beyond the culture of perpetual economic growth; this growth has largely been a 
blessing for several hundred years, but it has not brought us unalloyed happiness, and we 
must now choose between the pursuit of unrestricted and undirected growth through science 
and technology and the clear accompanying dangers. 
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It is now more than a year since my first encounter with Ray Kurzweil and John Searle. I see 
around me cause for hope in the voices for caution and relinquishment and in those people I 
have discovered who are as concerned as I am about our current predicament. I feel, too, a 
deepened sense of personal responsibility—not for the work I have already done, but for the 


work that I might yet do, at the confluence of the sciences. 


But many other people who know about the dangers still seem strangely silent. When pressed, 
they trot out the "this is nothing new" riposte—as if awareness of what could happen is 
response enough. They tell me, There are universities filled with bioethicists who study this 
stuff all day long. They say, All this has been written about before, and by experts. They 


complain, Your worries and your arguments are already old hat. 


I don't know where these people hide their fear. As an architect of complex systems I enter 
this arena as a generalist. But should this diminish my concerns? I am aware of how much has 
been written about, talked about, and lectured about so authoritatively. But does this mean it 


has reached people? Does this mean we can discount the dangers before us? 


Knowing is not a rationale for not acting. Can we doubt that knowledge has become a weapon 


we wield against ourselves? 


The experiences of the atomic scientists clearly show the need to take personal responsibility, 
the danger that things will move too fast, and the way in which a process can take on a life of 
its own. We can, as they did, create insurmountable problems in almost no time flat. We must 
do more thinking up front if we are not to be similarly surprised and shocked by the 


consequences of our inventions. 


My continuing professional work is on improving the reliability of software. Software is a tool, 
and as a toolbuilder I must struggle with the uses to which the tools I make are put. I have 
always believed that making software more reliable, given its many uses, will make the world 
a safer and better place; if I were to come to believe the opposite, then I would be morally 


obligated to stop this work. I can now imagine such a day may come. 


This all leaves me not angry but at least a bit melancholic. Henceforth, for me, progress will be 


somewhat bittersweet. 


https://www.wired.com/2000/04/joy-2/ 29/33 


6/19/2021 Why the Future Doesn't Need Us | WIRED 
Do you remember the beautiful penultimate scene in Manhattan where Woody Allen is lying 
on his couch and talking into a tape recorder? He is writing a short story about people who 
are creating unnecessary, neurotic problems for themselves, because it keeps them from 


dealing with more unsolvable, terrifying problems about the universe. 


He leads himself to the question, "Why is life worth living?" and to consider what makes it 
worthwhile for him: Groucho Marx, Willie Mays, the second movement of the Jupiter 
Symphony, Louis Armstrong's recording of "Potato Head Blues,” Swedish movies, Flaubert's 
Sentimental Education, Marlon Brando, Frank Sinatra, the apples and pears by Cézanne, the 


crabs at Sam Wo's, and, finally, the showstopper: his love Tracy's face. 


Each of us has our precious things, and as we care for them we locate the essence of our 
humanity. In the end, it is because of our great capacity for caring that I remain optimistic we 


will confront the dangerous issues now before us. 


My immediate hope is to participate in a much larger discussion of the issues raised here, with 
people from many different backgrounds, in settings not predisposed to fear or favor 


technology for its own sake. 


As a Start, I have twice raised many of these issues at events sponsored by the Aspen Institute 
and have separately proposed that the American Academy of Arts and Sciences take them up 
as an extension of its work with the Pugwash Conferences. (These have been held since 1957 


to discuss arms control, especially of nuclear weapons, and to formulate workable policies.) 


It's unfortunate that the Pugwash meetings started only well after the nuclear genie was out of 
the bottle—roughly 15 years too late. We are also getting a belated start on seriously 
addressing the issues around 21st-century technologies—the prevention of knowledge- 


enabled mass destruction—and further delay seems unacceptable. 


So I'm still searching; there are many more things to learn. Whether we are to succeed or fail, 
to survive or fall victim to these technologies, is not yet decided. I'm up late again—it's almost 
6 am. I'm trying to imagine some better answers, to break the spell and free them from the 


stone. 
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l The passage Kurzweil quotes is from Kaczynski's Unabomber Manifesto, which was 
published jointly, under duress, by The New York Times and The Washington Post to attempt 
to bring his campaign of terror to an end. I agree with David Gelernter, who said about their 


decision: 


"It was a tough call for the newspapers. To say yes would be giving in to terrorism, and for all 
they knew he was lying anyway. On the other hand, to say yes might stop the killing. There 
was also a chance that someone would read the tract and get a hunch about the author; and 


that is exactly what happened. The suspect's brother read it, and it rang a bell. 
"I would have told them not to publish. I'm glad they didn't ask me. I guess." 
( Drawing Life: Surviving the Unabomber. Free Press, 1997: 120.) 
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Penguin, 1994: 47-52, 414, 419, 452. 


3 Isaac Asimov described what became the most famous view of ethical rules for robot 
behavior in his book I, Robot in 1950, in his Three Laws of Robotics: 1. A robot may not injure 
a human being, or, through inaction, allow a human being to come to harm. 2. A robot must 
obey the orders given it by human beings, except where such orders would conflict with the 
First Law. 3. A robot must protect its own existence, as long as such protection does not 


conflict with the First or Second Law. 


4 Michelangelo wrote a sonnet that begins: 


Non ha l' ottimo artista alcun concetto * 
Ch' un marmo solo in sè non circonscriva 
Col suo soverchio; e solo a quello arriva 
La man che ubbidisce all' intelleto. 


Stone translates this as: 
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The best of artists hath no thought to show * 
which the rough stone in its superfluous shell 
doth not include; to break the marble spell 

is all the hand that serves the brain can do. 


Stone describes the process: "He was not working from his drawings or clay models; they had 
all been put away. He was carving from the images in his mind. His eyes and hands knew 
where every line, curve, mass must emerge, and at what depth in the heart of the stone to 


create the low relief." 
(The Agony and the Ecstasy. Doubleday, 1961: 6, 144.) 
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percent, among all humans who will ever have lived. This would be some reason for thinking 
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that humankind will not survive for many more centuries, let alone colonize the galaxy. 
Carter's doomsday argument doesn't generate any risk estimates just by itself. It is an 
argument for revising the estimates which we generate when we consider various possible 


dangers." (Routledge, 1996: 1, 3, 145.) 
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